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ABSTRACT ' . ' 

An evaluation design for educational eguity materials 
was utilized .with seven instructional units on sex rcles' and sex 
stereotyping for teachers and teacher educators* The instructional 
units were developed by the Project on Sex Stereotyping in Education 
in the areas of human growth ^nd development, matheaatics education, 
science educaticii, language arts education, -educational history, 
social studies education, and physical education* Each unit was 
reviewed by experts in sex stereotyping, subject area, and 
aulti-cultur al education. In embarrassment field tests, education 
/students completed questionnaires identifying possible unit 
improvements, Multiple-choice cognitive tests incorporated 
alternatives to' stereotyped behaviors anov ojths about aales and 
females* *An adaption of the Bern Sex ficle Inventcry (BSBI) , in which 
subjects rate themselves on characteristics which t are given 
masculinity, femininity, or androgyny scores, was used ^as an 
affective instrument for unit evaluations* l.he units and evaluation 
instruments were administered' by teacher educators* Results indicated 
that the, units significantly increased subjects 9 knowledge, but did 
littl^ to affect scores on the BSBI* Inclusion of periodic revision 
points* and the incorporation of expert and user reviews are among 
the strengths of the evaluation design* Design weaknesses include the 
difficulty of obtaining affective measurement results v whioh are both 
falid and reliable* ( Author /JAC) 
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This presentation will cover the role that evaluation can play in the 
development of multi-media instructional units on sex roles and sex stereotyping 

\ 

An evaluation design for the development of n sex role materials, whicty -includes 
content reviewers, multi-cultural reviewers, embarassment fdeld tests and 
cognitive, affective and observational validation components will be discussed. 
The strengths and weaknesses of the design, as found by its Application to the 
seven instructional units developed by the Project on Sex^Stereotyping in Educa- 
tion will also be discussed. Some of the design strengths that will be covered, 
include the inclusion of periodic revision points and the incorporation of a 
variety of .expert and user reviews. Design weaknesses include the difficulty, 
of getting affective measurement results which are both valid and reliable. 



y A Useful Evaluation of Sex Role Materials: ». 

*- ' Possibility or Pipe Dream?' * s 

— ■* • 

In the past £ew years there has fceen a tremendous upsurge in the development 

of educational materials on women's educational equity. The Federal Goveriment 

(WEEAP Final Report, 1976; WEEAP Final Report, 19,77), State Departments of Educa- 

** * * 

tion (Pennsylvania Department of Education, 1974; Pennsylvania Department at 

* s 

Education, 1973), school systems (Berkeley ^Unified School District, 1976; Ceres 
Unified School District, 1977), teacher organizations (Resource Center on Sex Rolfcs, 

1974; American Federation of Teachers, 197 A) , private organizations (Emma Wilfard 

/ 

Task 'Force, 1972; Rosen&eld - The Feminist Pr;ess, 1§76) and even individuals 

/ 

(Rothchild, 1973; Campbell and Thompson, 1974, Campbell, 1975) are developing and 
producing a wide range of materials whose, goal is to promote educational equity 

'4 - 

for males and females. 

Unfortunately many of these materials are not being evaluated. The reasons 
for this are^centered around three major factors - lack of money, lack of confi- 
dence in the evaluative process and lack of valid reliable measuring instruments. 

Many materials development efforts ajre funded only for supplies or not at all, 
with the developers frequently being people who volunteer their time because they 
feel there is a need for educational equity materials. Under these circumstances 
it is somewhat unreasonable to expect a^well designed, somewhat expensive evalua- 
tion. Even evaluation that could be done inexpensively by the developers themselves 

is frequently not done because of a feeling th^at all of the limited resource,^ avail- 

_> 

able* should be put into the materials. 

Developers disinclination « towards evaluation is' also because of a lack of 

/ i 

confidence in the evaluative process. Evaluators are often viewed as villains 

s 

whose rols is to hurt rather than help. (Granger and Campbell, 1977.) .Frequently, 
they are seen- as members of an educational establishment that has in the past 
shown little concern for educational equity. v 

4 • I 



Another major concern is measurement. Particularly in the affective varea, we 

/ 

do not know hoft to measure the effects tha< educational equity materials or social 
change materials in general^ ai/e having* And without' adequate measures eveor the 
best evaluation is doomed to failure. / 

These concerns are legitimate and serious. Although paper and penc mst ru- 
ments on sex roles and sex bias 'abound, few deal specifically with education and 
even fewer have validity and reliability. (Findley et. al.) Also, many pro- 
fessional evaluators are not knowledgable about and sensit ivc* to the issues and 
concerns of women's educational equity; while many materials developers are not 
knowleSgable about areas like design, measurement and statistics. Even if the 
evaluator and d.eveloper get together, they frequently have no common ground on 
which to meet. 

However, in spite of these difficulties, evaluation must be done on educational 
equity materials. Potential users need to know if materials are effective, develop- 
ers need fro know how the materials can be improved and* last but not least, 
federally funded materials developers need to provide the "government with" evidence 

that project objectives have been met. 

% 

| • Any evaluation of educational equit^^niaterials must include both formative 
aYid summative components. In additie>n,^pfiile the evaluation must be theoretically 
sound, it must also be practical and feasible. The following is one educational 
equity- materials development project's attempt to develop and implement such an 
evaluation, 

* 

The Project^ on Sex Stereotyping in Education (POSSE) was funded by the Women's 
Educational Equity Act Program of the Office of Education. Its goal was to develop 
and validate seven, instructional units on sex roles and sex Stereotyping^ for 
teachers and teacher educators. 'The mu^ti-mediated units it developed consisted 
of an audio tape, transparency masters, handouts, a bibliography, an instruction 
sheet and a type script of the audio tape. The seven content areas that were < 
covered were^human growth and development, math education, science education, 
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language arts education, educational history* social studies* education and physical 

education. ' „ 

Project objectives were to create units that 

; !. provided up-to-date research information s"ex stereotyping and 
its effects-^ 

2. .refuted traditional myths about*the roles and activities of women 
v *v. - . an ^ men, bctyfc and girls; 

3. increased teachers 1 awareness of their influence on the development 
or their 'students sex roles; - . , 

4. provide alternatives to stereotypic behavior for- teachers; 

5. 'could be used independently or adapted by course instructors 
, (-Campbell," 1976). > 

An evaluation design was developed that sought to determine if the project 
/ 

met its objectives and what effects, if any, the units bad on pre- and in-service 
teachers. t 

The* formative evaluation component involved the use of reviewers arid fseld 
tests. Each unit was reviewed f©r accuracy and completeness by people with exper- 
tise in sex stereotyping, unit content area and multi-cultural education. For 
example, the math unit was reviewed by Dr. Dora Skypek, a math educator from Emory 
University ancj by Dr. Gwen Baker, Director of the Mult i-cultural- Educat ional Pro- 
gram at the University of Michigan. The review process 'and the revisions, based 
pn reviewer feedback, insured that accurate, complete and up-to-date information 
was provided by the unit. 

A second, component of the formative evaluation v^as the embarassment field 
test. £acti completed unit was given to a group of education students who upon 
completing the unit filled out an open-ended field test , questionnaire. The stu- 
dents were asked to list the things they liked most, the things thev liked least, 
and the ways they would improve the instructional unit as a whole, the audio tape, 
,the transparencies ahd the handouts. Students were £lso interviewed by project 
'staff to determine students 1 perceptions of the clarity, interest and usability 



C ■ " . .. ' ' 



pi the, unit.. Based^pn the responses to fche questionnaires and the interviews, each 
unit was revised. ' 

v . . ■ ' • . ' 

• A' final component of the formative evaluation was the suntmative evaluation of 
earlier units. That is, based on the results of the evaluation of the first 
several units, changes were made in the formats and development process of the 
later units. The three step process of review, field test and learning from pre- 
vious mistakes was found* by project staff to be ^oth useful and efficient u 
- The summative evaluation was centered around a national validation of the 

units using both cognitive and aff ective ' instruments. Each junit was . lis ted on a 

- n 

national sample of pre- and in-service teachers using -a pre-test treatment post- 

t 

test design. 

Cognitive tests were developed- for each unit using a tatfle of specifications. 
Test items were in multiple choice and true-false formats and included questions 
on myths about females and males and on alternatives to stereotypic behaviors. 
Items for each unit cognitive test were field tested by graduate education stu- 
dents. Based on an item analysis, project director review and the table of speci- 
fications, twenty to thirty items were selected for each unit test. Each cognitive 
test was also analyzed for internal consistency. The results ranged from a low of 
.59 for the science cognitive test to a high of .88 for the social studies test. 

Developing an affective test for the units was a difficult task. A search 

of the literature for appropriate existing instruments proved fruitless, as did \ 
■ * . * 

an attempt byproject staff to construct a valid, reliable measure. As a result, 

an adaptation of the Bern Sex Role " Inventory (BSRI) was used as a^pre and post. 

measure. In the BSRI, subjects rate themselves on a series of characteristic*^ 

Eased, on their ratings, three scores are calculated; a masculinity score , a 

femininity score and an androgyny score. Originally the adaptation was to have 

subjects rate a boy or a £irl on a series of characteristics"; however, our own 

research indicated that the age level of the boy or girl was a major fac,tor x in 



subject's ratings. Therefore, in six of the seven units, subjects' were 'asked to 

rate the characteristics of either arK elementary school girl, an elementary school 

boy, a high school girl or a/Kigh school boy. ' (For the seventh unit, on educational 

history, subjects were asked to rate either a male or female educator*because, 

unlike the previous units, its content dealt with adults rather than children.) 

^Tfre BSRI was selected for use bepause of its high reliability (.93, Bern, 1^74) and 

its criterion validity (Bern, 1975). 
s ^- 
Using the adapted BSRl and the cognitive unit tests, the summative evaluation 

was done with -the assistance of an eleven member national task force of teacher 

educators, , ^ 1 

For each unit, participating task force members were sent an evaluation packet- 
that included* instruct ions, copies offtthe complete unit and testing sets. The 
testing s*ets were composed of instructions, pre-af f ective and cognitive test-s and 
post-cognitive and affective tests. 

, Task force members were instructed to give the ;pre-tests , have the subjects go 
through the unit and give the post-tests.. Interviews with the task force members 
indicated they found no difficulties in giving either the units or the tests. 

Each unit was used by 'at least five people with no previous experience in 
working with the unit. In each case, without consultant assistance, the people were 
able to use the unit successfully. In two cases, people were asked to set up the 
units without any prior instruction and in both cases* they were successful. * 

Originally it was planned to have the post-tests given a week after the stu- 
dents had completed' the unit. Unfortunately, it was discovered' that this procedure 
cut the number of available' subjects down considerably. Therefore .the post-tests 
were given immediately after the unit was completed. All reported subject scares 
are on post-tests taken immediately after completing the unit. 

The results* o'f the cognitive tests were combined^ 1 totaled and analyzed by the\ 
school. If a School sample size was under twenty^ a related t test was used, if the 
sample si2e was x>ver twenty, a g^test was used! - Only one of the analyses (one 
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schcfcl in the human growth and development unit sample) was foiind to be non- 

significant. In over 80% of the analyses, the pest-test was found to be signifi- 

^ <- 
cantly higher than the pre-test at the .0005 level.. 

Table 1 gives a summary of the mean pre and post cognitive tests scores in 

percentages and the Z scores for the seven units. 

Table 1 



A Summary of Cognitive Test 


Results - 


■ 






« 

Human Growth and Development Unit 


Pre 
54 


Post ' 
70 


Z = 


z 

13.57* • 


N 
322 


Matfh Unit 


60.7 


73.8 


Z = 


7.7* 


1-61 


Science Unit 


54.6 


72. 1 


z = 


11.8* 


131 


Educational History Unit *^ 


48.5 


75.6 


z = 


22.7* 


193 


Language Arts Unft 


66.8 


78.3 


z = 


8.85* 


153 


Social Studies Unit 


60 ■ 


75 


2 = 


10. 79** 


155 


Physical Education Unit 


58.8 


73.9 


z = 


9.21* 


130 



* p . y)005 

The affective androgyny scores from ,the science, math, educat ional' history, 
language arts, social studies and physical education units were analyzed using an 
analysis of variance with repeated measures with equal cell sizes. Data was 
randomly eliminated from some cells in order to get equal celf sizes*. 

The human growth and development data was analyzed using analysis of'varianoe 
This datai was not able to be analyzed using an analysis of variance with repeated 
measures because many of the subjects did not follow directions and did not write 
their subject numbers on both the pre and the post tests. In order to alleviate 
this problem in the other units, the rest of the validation packets were sent out 
with subject numbers already stamped on the^pYe and post tests. 



In the human growth and development unit affective test results showed signif 
cant F scores on the treatment (pre-post), sex^>female--male) and*level (elementary 
high school) variables. However post hoc art^Uysis^ found no significant change at 



th6 .05 level in 1 the ratings for. elementary and high school boys and girls before 
and'after going through the unit. > . 

Table' 2 

Human Growth and Development Unit * 

.Affective Test Results it ' ' 



Boy 
Girl 



Elementary 


Pre 

High School 








Post 

Elementary High School 


- 1.87 


- 2.56 






. Boy 


- 2.00 - 2.9$ 1 

- .i 


+ 2.25 


+ 2.11 




4 


Girl 


1.92 . * 1.23" " | 



Source of Variation 



Pre-Post (P) 

Elementary-High School (E) 
Male-Female (S) 
P x E 

i 

P x S 
E x S 
P x E x S 
Error 



Sum of Squares 
31.5119 
64.8951 
2917'. 9895 
7.4450 
4.4590 
7.4649 
& 0.5972 
3766.9600 



D.F. 



650 



Mean Square 
31.5119 
64.8951 
2917.9895 
7.4450 
4.4590 
7.4649 
0.5972 
5.7953 



5T437 5 * 
11 . 1978, ** 
503.507 6 ** 
1. 2847 
0.7694 
1.2881 
0. 1030 



// Scores approaching zero are androgynous-. 
Scores approaching +4 are feminine. 

Scores approac,hing 4 -4 are masculine. ' 

* p < .025 
** p < .'001 

In the math unit, affective test results showed significant F scor. s on the 
variables, ^ sex, level ( elem'entary-high school) and treatment (pre-post) . The 
interaction of sex, level and treatment was also significant. Post hoc analysis 
found no significant changes at the .05 level for the ratings for elementary and 
high school girls. However, elementary and high school boys were rated si^nifi- 
cantly more masculine after subjects took the unit. 



„ Table 3 

Math Unit 
♦Affective Test Results // 
N=30 cases per cell 





Pre • 
















School 












Elementary # High 






Elementary Hi 


gh School 


Boy 


- .97 


3.19 ] 


i 


>Boy 


- 1.71 


- 3.93 


Girl 


+ ,2.77 +' 


2.03 j 




Girl 


+ 2.80 


+ 2. OA 


Source of Variation 


" * Sum 


of ^fl 11 3 T"£»^ 
\J ± JUUCIL CO 


D. F. 


Moan ^fiiiarp 


F. 


Elementary-High School 


(E) 


J- J (- « *T \J 


i 
j. 




1 \ OR ** 

J. J « wo 


Male- 


Female (S) 




141 / . J 


1 


1 A 1 7 ^ 4 


1 TO Q7 


E x S 














Error 






1174.63 


JL16 


10.13' 




Pre-post (P) , - 




7.8 


1 


7.8 


A. 20 * 


E x P 




« 


.00305 


1 


.00305 ' 


1 /0016 


S x P 






8*81 


1 


8.8L 


A. 74 * _ 


E x S 


x P . 




.00717 - 


1 


.00*17 


.0039 


Error 




215.5 


116" 


1.86 





// Scores approaching zero are androgynous.- 
Scores approaching +4 are feminine. 
Scores approaching -A are masculine. 

* p < .05 ; 

**- p < .001 

In the science unit, affective test results showed a significant F score on 
the variable sex. No significant differences on the variables level (elementary- 
higtt school) or treatment (pre-post) nor any significant interactions were found 
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' Table 4 




s m * 

. ..) ' 




r 


\ 


' . Science Onit 

Affective Jest Results // 
'N-29 cases per cell 

' • • { • * 


1 1 


1 

* 


/ 




Pre 

Elementary High School 




/ 


Post • 
Elementary HigTi School 






Boy 


- 0.70 - 2103 




Boy 


- .92 - 2.13 


• 


• 


Girl 


* 1.49 - - + 1.3*4 




r G±tl) 


+ 1.54 + 1.57 






Source of Variation Sum of Squares 


D.F. 


Mean Square ' F. 






Elementary-Secondary (E) 


25.74 


1 


*25.74 1.92 


• 




Female-Male (S) 


495.33 ~ 


1" * 


495.33 37,00 * 






E x 6 




■v 

21.26 


1' 


21.26 1.58 






,Error 




1499.26- 


112 


13.34 


♦ 


* 


Pre-Post (P) 
P x E 


.004 
.29 


1 

1 


» 

" - .004 .002 
.29 " .1-5 


* f 




P x S^ 


1.31 


1 


S 1.31 ' .70 






*P x E 


x S 


.01 


1 


.01 , .007 






Error 




209.83 


112 


1.87 • 




s 














i 


// Scores approaching zero are androgynous* 
Scores approaching +4 are feminine. 
Scores approaching -4 are* masculine/ 






• 




* P < .05 , 

♦ 
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> 


♦ « * 
In t,he educational history unit, a-ffective test results showed a significant 


- 


F score on the variable sex. Nb significant differences on th*e variable treat- 

ment (pre-post) nor any significant interactions between sex and treatment were 

'f * 






found 
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- • 










• 
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• 
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Table 5 

\ -Educatiorial History 
Affective Tdst Results : 
' N=67 cases per cell • 



Male Educator 
Female Educator 




ource of Variation 



Sum of Squares D.F. 



Male-Female (S) 



Error 

Pre-Posft (P) 
P x s\ 
Error 1 



1154.49 
1097.18 

,'.073 
1.41 
98.89 



v. L 
132 

1 
"1 

132 



Mean Square 
1154.49 
8.31 

.07 
1.41 
*' • .074 



138.89 * 

.09 
' 1.-89 



// Scores- approaching zero are androgynous. m A 

Scores ^approaching +4 are feminine. • % 

. Scores approaching -4 are masculine. * 

*p <;.ooi ♦ - " •* / 

In the language arts unit, effective test 'results showed significant F scores 
on the_ variables sex, level (elem^ntary-Jhigh school) and treatment t (pre-post) # an d 

en*the interaction of sex, level and treatment. Post hoc analysis found rto signi- 

"* 

f icant changes at the .05 level for the ratings for elementary and hig^school 

• v . 
,girl£ and for kigh»school "boys. However elementary boys were rated significantly 

more masculine after subjects took the unit. 



X 



^1 




Table« 6 



Language Arts Unit 
Affective Test -Results // 
N=34 cases per cell 



Pre * 



Post 





Elementary 


' High School 




* 


• 


Elementary 

■ 


High School 


Boy 


— « 

- i.oe - 


- 2.26 i 

t 






Boy , 


- 2. OA • 


- 2.83 


Girl'' 


+ 2..2A '. 


1 

+ 1.39 






Girl 


- 2.71 


.+ 0.9 



Source' of Variation 



Elementary-Secondary. (E) 



Male-Female (S") 



E x S 
Error 

Pre-Post (P) 

P x E 

P x S 

P x E. x S 

4 

// Scores approaching zero are androgynous. 
Scores approaching +A are- feminine . 
Scores approaching -A are masculine. 

* p < .05 
'** p < .001 




Q-F. 
■ 1 

1 

1 

132 

1 
1 

1 • 
1 



Meap Square 

91.69 
1009. 6A 

1.89 

9.61 , 

10.2$ 
1.31 

"9.81 
8i08 



9.5A * 
105.05 ** 

.19 '' 



A.2A * 

.54 
A. 06 * 
3.3A 



In the social studies unit, affective test results showed a significant F 
score on the variable sex. No significant differences on the variables level 
(elamentary-high ^school) or treatment (pre-post) nor any significant interactions 
were found. 
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- 


* 


4 


t 

T a K1 cs 7 * 
Idulc / 

Social Studies Unit 
Affective Test Results # 
J N=25 cases per cell 


• 








Pre 






) 


* 

Post 






- 


Elementary High School 






Elementary High School 


* 


Boy 


1.A5 - 1. 


81.\ 




Boy 


- 1. 51 


1 

- 2. -2 




. *Girl 


* '+ 2. AO + 1. 


51 




Girl 


+ 1.95 


+ 1.27 




















Source 


of Variation 


Sum 


of Squares S 


D.F. 


Mean Square 






Elementary-Secondary (E) 




21.60 


. 1 


21.60 


2.17 
** 




Male-Female (S) 




620.49 


1 


620.49 


62.56 * 




E< x S 






0.90 


. -1 


" 0.90 


0.09 




Error 






952.02 > 


96 


9.91 






Pre-Po<5t (P) . 




A. 035 


1 


A. 035 


3.25 




P x E 






0.04 


1 


1 A A/, 

0 . OA 


0.03 




P x S 






0.18 . 


1 ' 
L 


0.18 


0.14 


• 


P x E< 


x S 




0.93 


1 


0.93 


0.75 


• 


Error 






118.99 


96 


1.23 






// Scores approaching zero are androgynous 
Scores approaching +A are feminine. 
Scores approaching -A are masculine. 




♦ 




* P<. 


001 














t 

In the physical education unit, affective test results showed s 


ignificant F 


\ 


' scores 


on the variables sex 


, level (elementary-high school) and treatment (pre-post). 


1 No -significant: interactions 


were 


found. Post 


hoc analysis found no 


significant 




changes at the 05 level for the ratings for 


elementary <and high school girls and for * 


elementary boys. However, 


high 


school boys were rated significantly 


more masculine ' ^§ 


- after subjects took the unit* 

ERIC 


1C 








• 
• 



Table 8 « 

■ Physical Education Unit 
Affectiv'e Test. Results // 







N 


-28 cas'es per ,cell 










f Pre 
Elementary High 


f 

School 


/■ 

f 




Post 

Elementary High 


School 




Boy 


- .89 


<2.40-„ 




Boy 


- 1.14 


3.04 




Girl 


+ 2. 17 + 


i'w*r>. 




Girl 


f 1.98 + 


i.oo . 


i 


Source 


of Variation 


Sum of ^Squares 


D.F. 


Mean Square 
100.94 


F\_ . 




Elementary-Secondary (E) 




100.94 


s 1 


12.26 


* 


Male-Female (S) 




" 667.37 


1 


667.37 


81.06 


** 


E x S 






• '7.14 • 


1 


7.14 


.86 




Error 




s 


889.07 


108 


8.23 ' 







. Pre-Post (P) 
P x E 
P x S 
P x E x S 
Error 



5 .'46 
.49 
' .96 
" • .58 
102.82 



1 
1 
1 
1 

103 



5.46 
.49 
.96 
.58 
.95 



5.73 * 

.51 
.1.01 
.61 



// Scores approaching zero are androgynous. 
Scores approaching +4 .are feminine. 
Scores approaching -4 are masculine. 



* p < .05 
** p < .001 



r 
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Overall there were^few pre-post differences in the affective results. The 
four significant difjjg^enc^es that wer$' found indicated a fendency to rate^boys 
more masculinfe on the post-t;est tharf ort the pre-test: 

These results must be } viewed with caution for several reasons. First, a 
number of subjects seemed offended by the affective questionnaire. About 10%. of 



IC 



r 



the subjects* did not complete, the affective tests while less than 1% of t?he subjects 

did not complete, the cognitive tests. Several peopLe wrote comments indicating dis- 

satisfaction with the affective test. For example, one person wrote "The tape tells 

us we shouldn't stereotype but then you give us a test that forces us to stereotype. 

^ , <&> 
Other people completely filled out the test with the neutral response. 

It may be that the math, language** arts an<4 physical education units (those* 

units in which a significant difference was found) included information about the . 

characteristics of boy students* that mate the subjects view boys as possessing a 

greater degree of masculine characteristics. It may also be that, as one subject 

wrote "This test has nothing* to do with' the goals of the tape.", and the results 

tell us little about the tapes. 

^ Also recent research indicates that changes in teachers stereotypic classroom 

"behavior may not be reflected on their scores on the BSRI. Redd (1976) found that 

attending several sessions on sex role stereotyping in education significantly 

reduced teachers stereotypic classroom behavior but did not affect their scores or> 

4 

the BSRI. 

In summary, the units significantly increased subjects 1 knowledge but did 
little to affect subjects' scores on the BSRI. 

For the; 1-977-78 year, POSSE has retained many of the evaluative components 
but is incorporating a classroom observation component albng with a paper and 
pencil affective measure. However, the cost and difficulty of conduct L.g 'observa- 
tions preclude the widespread use of this method. More research and developfnent 
j * 

work is needed on ways of measuring the impact of social change materials in the 
affective domain. Until that work is done, a useful evaluation of sex role 
materials must remain more a pipe dream than a possibility. 
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